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Abstract:  An  interpolation  method  is  presented  for  the  reconstruction  and  recognition  of  human  face  images.  Basic 

ingredients  include  an  optimal  basis  set  defining  a  low-dimensional  face  space  and  a  set  of  “best  interpolation 
pixels”  capturing  the  most  relevant  characteristics  of  known  faces.  The  best  interpolation  pixels  are  chosen  as 
points  of  the  pixel  grid  so  as  to  best  interpolate  the  set  of  known  face  images.  These  pixels  are  then  used  in  a 
least-squares  interpolation  procedure  to  determine  interpolant  components  of  a  face  image  very  inexpensively, 
thereby  providing  efficient  reconstruction  of  faces.  In  addition,  the  method  allows  a  fully  automatic  computer 
system  to  be  developed  for  the  real-time  recognition  of  faces.  Two  significant  advantages  of  this  method  are: 
(1)  the  computational  cost  of  recognizing  a  new  face  is  independent  of  the  size  of  the  pixel  grid;  and  (2)  it 
allows  for  the  reconstruction  and  recognition  of  incomplete  images. 


1  INTRODUCTION 

Image  processing  and  recognition  of  human  faces 
constitutes  a  very  active  area  of  research.  The  field 
has  evolved  rapidly  and  become  one  of  the  most  suc¬ 
cessful  applications  of  image  analysis  and  computer 
vision  partly  because  of  availability  of  many  power¬ 
ful  methods  and  partly  because  of  its  significant  prac¬ 
tical  importance  in  many  areas  such  as  authenticity 
in  security  and  defense  systems,  banking,  human- 
machine  interaction,  image  and  multimedia  process¬ 
ing,  psychology,  and  neurology.  Principal  component 
analysis  (PCA)  or  the  Karhunen-Loeve  (KL)  expan¬ 
sion  is  a  well-established  method  for  the  representa¬ 
tion  (Sirovich  and  Kirby,  1987;  Kirby  and  Sirovich, 

1990)  and  recognition  (Turk  and  Pentland,  1991)  of 
human  faces. 

PCA  approach  (Kirby  and  Sirovich,  1990)  for  face 
representation  consists  of  computing  the  “eigenfaces” 
of  a  set  of  known  face  images  and  approximating  any 
particular  face  by  a  linear  combination  of  the  leading 
eigenfaces.  For  face  recognition  (Turk  and  Pentland, 

1991) ,  a  new  face  is  first  projected  onto  the  eigenface 
space  and  then  classified  according  to  the  distances 
between  its  PCA  coefficient  vector  and  those  repre¬ 


senting  the  known  faces.  There  are  two  drawbacks 
with  this  approach.  First,  PCA  may  not  handle  cor¬ 
rupted  data  well,  that  is,  situations  in  which  only  par¬ 
tial  information  of  an  input  image  is  available.  Sec¬ 
ondly,  the  computational  cost  per  image  classification 
depends  on  the  size  of  the  pixel  grid.  Despite  this, 
PCA  is  still  one  of  the  most  used  techniques  for  face 
recognition  due  to  its  simplicity  and  efficiency  over 
other  methods. 

This  paper  describes  an  interpolation  method  that 
aims  to  address  these  deficiencies  of  PCA.  The 
method  was  first  introduced  in  (Nguyen  et  al.,  2006) 
for  the  approximation  of  parametrized  fields.  Here, 
we  investigate  the  method  for  face  reconstruction  and 
recognition.  The  basic  ingredient  is  a  set  of  “best 
interpolation  pixels”  capturing  the  most  relevant  fea¬ 
tures  of  known  face  images.  The  essential  component 
is  a  least-squares  interpolation  procedure  for  the  very 
rapid  computation  of  the  interpolant  coefficient  vec¬ 
tor  of  any  given  input  face.  The  interpolant  coeffi¬ 
cient  vector  is  then  used  to  determine  which  face  in 
the  face  set,  if  any,  best  matches  the  input  face.  A  sig¬ 
nificant  advantage  of  our  approach  is  that  the  compu¬ 
tational  cost  of  recognizing  a  new  face  is  independent 
of  the  size  of  the  pixel  grid,  while  achieving  a  recog- 
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nition  rate  comparable  to  PCA  approach.  Moreover, 
the  method  allows  the  reconstruction  and  recognition 
of  corrupted  images. 

In  the  past  years,  there  have  been  a  large  number 
of  contributions  to  face  recognition  and  analysis.  The 
reader  is  referred  to  a  number  of  papers  (Chellappa 
et  al„  1995;  Jain  and  Li,  2005;  Delac  et  al„  2005; 
Draper  et  al.,  2003;  Phillips  et  al.,  1998)  for  perspec¬ 
tives  and  recent  advances  in  face  recognition.  Face 
analysis  and  representation  have  also  been  extensively 
studied  by  many  authors  (Sirovich  and  Kirby,  1987; 
Kirby  and  Sirovich,  1990;  O’Toole  et  al.,  1993;  Ever¬ 
son  and  Sirovich,  1995;  Kanade,  2005). 

This  paper  is  organized  as  follows.  In  Section  2, 
we  present  an  overview  of  PCA.  In  Section  3,  we  ex¬ 
tend  the  best  points  interpolation  method  (BPIM)  in¬ 
troduced  in  (Nguyen  et  al.,  2006)  and  apply  it  to  de¬ 
velop  an  automatic  real-time  face  recognition  system. 
In  section  4,  we  test  and  compare  our  approach  with 
PCA.  Finally,  in  Section  5,  we  close  the  paper  with 
some  concluding  remarks. 


2  PRINCIPAL  COMPONENT 
ANALYSIS 

2.1  Eigenfaces 

An  ensemble  of  face  images  is  denoted  by  Uk  =  {«, }, 
1  <i<K.  where  u\  represents  an  /-th  mean-subtracted 
face  and  K  represents  the  number  of  faces  in  the  en¬ 
semble.  It  is  assumed  that  after  proper  normaliza¬ 
tion  and  resizing  to  a  fixed  pixel  grid  S  of  dimen¬ 
sion  Ai  by  Nj,  Ui  can  be  considered  as  a  vector  in  an 
A-dimensional  image  space,  where  N  =  N\Ni  is  the 
number  of  pixels.  PCA  (Sirovich  and  Kirby,  1987; 
Kirby  and  Sirovich,  1990)  derives  an  optimal  repre¬ 
sentation  of  the  face  ensemble  in  the  sense  that  the 
average  reconstruction  error 

K  k  2 

e*  =  X  ui-  “Of/  ,  (!) 

1=1  7=1 

is  minimal  for  all  k  <  K.  In  the  literature  (Turk  and 
Pentland,  1991),  the  basis  vectors  (|) ;  are  referred  as 
eigenfaces  and  the  space  spanned  by  them  is  known 
as  the  face  space.  The  construction  of  the  eigenfaces 
is  described  as  follows. 

Let  U  be  the  N  x  K  matrix  whose  columns  are 
[mi  , . . . ,  uk].  It  can  be  shown  that  the  <),■  satisfy 

A<t>;  =  ,  (2) 

where  the  covariance  matrix  A  is  given  by 

A  =  2uur  .  (3) 
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Figure  1 :  Eigenfaces  and  the  mean  face.  The  mean  face  is 
on  the  top  left  and  followed  by  1 1  top  eigenfaces,  in  order 
from  left  to  right  and  top  to  bottom. 


Here  the  eigenvalues  are  arranged  such  that  X  i  >  . . .  > 
Xk ■  Since  the  matrix  A  of  size  N  x  N  is  large,  solving 
the  above  eigenvalue  problem  can  be  very  expensive. 

However,  if  K  <  N,  there  will  be  only  K  meaning¬ 
ful  eigenvectors  and  we  may  express  any  <|>,-  as 

K 

<t><  =  Y  Wijuj  ■  (4) 

7=1 

Inserting  (3)  and  (4)  into  (2),  we  immediately  obtain 

Gtp,  =  Xityi  ,  (5) 

where  G  =  ^UrU  is  a  symmetric  positive-definite 
matrix  of  size  K  by  K.  The  eigenvalue  problem  (5) 
can  be  solved  for  tp,y,l  <  i,j  <  K .  from  which  the 
eigenfaces  ( \>j  are  obtained. 

2.2  Face  Reconstruction 

We  briefly  describe  the  reconstruction  of  face  im¬ 
ages  using  PCA  and  later  compare  the  results  with 
those  obtained  using  our  method.  To  this  end,  we 
seek  to  project  an  input  face  u  onto  the  face  space 


®A.  =  span{(|)i,.. 

.  ,(|)A}  to  obtain 

k 

u*  =  Yu  a‘  > 

1=1 

(6) 

where  for  i  =  1 , . 

■■,k, 

ai  =  (f>f  a  . 

(7) 

We  also  define  the  associated  error  as 

8*  =  || M  —  M* ||  . 

(8) 

Note  that  the  mean  face  of  the  ensemble  Uk  should 
be  added  to  u*  to  obtain  the  reconstructed  image;  and 
that  if  k  is  set  equal  to  K.  the  reconstruction  is  exact 
for  all  members  of  the  ensemble. 

We  present  in  Figure  1  the  mean  face  and  a  few 
of  the  top  eigenfaces  for  a  training  ensemble  of  400 
face  images  extracted  from  the  AT&T  database  (see 
Section  4.1  for  details).  Figure  2  shows  8*el  as  a  func¬ 
tion  of  k.  Here  e*e]  is  the  average  of  the  relative  error 


Figure  2:  Average  relative  error  e*ej  versus  k  for  the  training 
ensemble. 


8*/ 1|  u  ||  over  the  training  ensemble.  Note  that  the  slow 
convergence  is  mostly  due  to  the  wide  and  complex 
variation  of  faces  in  the  ensemble.  Much  faster  con¬ 
vergence  and  smaller  error  can  be  achieved  by  only 
considering  the  frontal  view  and  the  oval-shaped  por¬ 
tion  for  the  faces  (Kirby  and  Sirovich,  1990). 

2.3  Face  Recognition 

We  briefly  describe  the  eigenface  recognition  proce¬ 
dure  of  Turk  and  Pentland  (Turk  and  Pentland,  1991). 
To  classify  an  input  image,  one  first  obtains  PCA  co¬ 
efficients  a;.  1  <i<  k.  as  described  above.  One  then 
computes  the  Euclidean  distances  between  its  PCA 
coefficient  vector  a  =  [a i,...1a/(\T  and  those  repre¬ 
senting  each  individual  in  the  training  ensemble.  De¬ 
pending  on  the  smallest  distance  and  the  PCA  recon¬ 
struction  errors*,  the  image  is  classified  as  belonging 
to  a  familiar  individual,  as  a  new  face,  or  a  non-face 
image.  Several  variants  of  the  above  procedure  are 
possible  via  the  use  of  a  different  classifier  such  as  the 
nearest-neighbor  classifier  and  a  different  norm  such 
as  L\  norm  or  Mahalanobis  norm  (Delac  et  al.,  2005). 

It  is  generally  observed  that  the  recognition  per¬ 
formance  is  improved  when  using  a  larger  k.  Typ¬ 
ically,  the  number  of  eigenfaces  k  required  for  face 
recognition  varies  from  (9(10)  to  (9(102)  and  is  much 
smaller  than  N.  We  note  that  classification  of  an  in¬ 
put  image  requires  the  evaluation  of  PCA  coefficients 
according  to  (7).  The  computational  cost  per  image 
classification  is  thus  at  least  0(Nk).  This  cost  depends 
linearly  on  N  and  is  quite  acceptable  for  a  small  num¬ 
ber  of  input  images.  However,  when  classification  of 
a  very  large  number  of  faces  is  performed  at  the  same 
time,  PCA  approach  appears  increasingly  intractable. 
Real-time  recognition  is  thus  excluded  for  large-scale 
applications.  Other  subspace  methods  such  as  in¬ 


dependent  component  analysis  (ICA)  (Draper  et  al., 
2003;  Bartlett  et  al.,  2002)  and  linear  discriminant 
analysis  (LDA)  (Etemad  and  Chellappa,  1997;  Lu 
et  al.,  2003)  suffer  from  similar  drawbacks. 


3  BEST  POINTS 

INTERPOLATION  METHOD 

In  this  section,  we  extend  the  best  points  interpo¬ 
lation  method  (Nguyen  et  al.,  2006)  to  deal  with  face 
images  and  apply  it  for  face  recognition.  We  shall  use 
the  eigenfaces  as  basis  functions  in  the  interpolation 
process,  as  they  possess  optimal  L2  representation  of 
face  images.  The  key  idea,  however,  is  to  find  a  set  of 
interpolation  points  which  provides  a  good  uniform 
approximation. 

3.1  Interpolation  Procedure 

Let  us  recall  the  pixel  grid  S  and  the  face  space 
®Z:  =  span{4>  1 , . . . ,  (|)/; }.  In  this  space,  we  shall  seek  an 
approximation  of  any  input  image  u.  However,  rather 
than  performing  the  projection  of  u  onto  O  /;  for  the 
best  approximation  it* ,  we  choose  to  interpolate  u  as 
follows. 

In  particular,  we  aim  to  find  a  good  approxima¬ 
tion  u  (r_  db  of  m  via  m(>  k)  interpolation  pixels  {/ j  £ 
H},  1  <  j  <  m,  such  that 

k 

u='£d$i  (9) 

i=l 

where  the  coefficients  a,  are  the  solution  of 

k 

^<\>i{zj)di  =  u(zj),  j=l,...,m.  (10) 

i=l 

We  define  the  associated  error  as 

8  =  || u  —  u||  .  (11) 

In  general,  the  linear  system  (10)  is  over-determined 
because  there  are  more  equations  than  unknowns. 
However,  the  interpolant  coefficient  vector  a  = 
[a\,. ...  ,dk]T  can  be  determined  from 

CrCa  =  Crc  ,  (12) 

where  C  €  Rmxk  with  Cp  =  <|>((zy),  1  <  i  <  k,  1  <  j  < 
m  and  c  =  [k(zi),  . . . ,  u(zm)]T .  It  thus  follows  that 

a  =  Be  .  (13) 

Here  the  matrix  B  =  (CrC)  '  C7  is  precomputed  and 
stored.  Therefore,  for  any  new  face  u,  the  cost  of 
evaluating  the  interpolant  coefficient  vector  a  is  only 
0{mk)  and  becomes  0(k 2)  when  m  =  0{k). 
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Figure  3:  Distribution  of  the  best  interpolation  pixels  on  the 
pixel  grid  for  k  =  100  and  m  =  200. 


Table  1:  Average  interpolation  error  erei  and  average  error 
ratio  Y  for  different  values  of  k  for  m  =  2k. 


Obviously,  the  approximation  quality  depends 
crucially  on  the  interpolation  pixels  {z  j}.  Therefore, 
it  is  extremely  important  to  choose  {z  j}  so  as  to  guar¬ 
antee  accurate  and  stable  interpolation.  For  instance. 
Figure  3  shows  the  interpolation  pixels  for  k  =  1 00 
and  m  =  200  obtained  using  our  method  described  be¬ 
low.  We  see  that  the  pixels  are  distributed  somewhat 
symmetrically  with  respect  to  the  symmetry  line  of 
the  face  and  largely  allocated  around  main  locations 
of  the  face  such  as  eyes,  nose,  mouth,  and  jaw.  Fur¬ 
thermore,  we  present  in  Table  1  erei  and  Y  as  a  func¬ 
tion  of  k  for  the  choice  of  m  =  2k.  Here  erei  and  Y  are 
the  average  of  the  relative  error  8/||m||  and  the  average 
of  the  error  ratio  e/e*  over  the  training  ensemble,  re¬ 
spectively.  The  average  error  ratio  Y  is  quite  close  to 
unity  for  all  k.  This  means  that  the  approximation  u  is 
also  close  to  the  “best”  approximation  u*.  However, 
unlike  ir  ,  u  requires  only  m  intensity  values  of  u. 

3.2  Best  Interpolation  Pixels 

We  proceed  to  describe  our  approach  for  determin¬ 
ing  the  interpolation  pixels.  The  crucial  observation 


is  that  much  of  the  surface  of  a  face  is  smooth  with 
regular  texture  and  that  faces  are  similar  in  appear¬ 
ance  and  highly  constrained;  for  example,  the  frontal 
view  of  a  face  is  symmetric.  Moreover,  the  value  of 
a  pixel  is  typically  highly  correlated  with  the  values 
of  the  surrounding  pixels.  Therefore,  a  large  number 
of  pixels  in  the  image  space  does  not  represent  phys¬ 
ically  possible  faces.  Only  a  small  number  of  pix¬ 
els  may  suffice  to  represent  facial  characteristics.  The 
question  we  aim  to  address  is  to  find  such  pixels  and 
proceed  with  our  interpolation. 

We  choose  the  interpolation  pixels  by  exploiting 
the  training  ensemble  Uk-  Specifically,  we  might 
consider  to  choose  { z  /}  by  formulating  a  minimiza¬ 
tion  problem  that  minimizes  the  sum  of  squared  er¬ 
rors  between  U(,  1  <  (  <  K .  and  their  approximations. 
More  precisely,  we  might  wish  to  find  {zj}  as  a  min- 
imizer  of 

K  k  2 

jiiin  _X  ue-'2jaa(xu...,xm)tyj  (14) 

X1€  =  ,...,XmG=f=]  i=y 

k 

X  <1 h(xj)au  =  ue(xj),  l<j<  m,  1  <£<K. 

i=  1 

Clearly,  the  minimizer  of  the  above  error  minimiza¬ 
tion  problem  is  optimal  for  the  interpolation  of  the 
face  images  belonging  to  Uk-  However,  since  the 
problem  is  nonconvex  with  multiple  local  minima  and 
the  Hessian  is  not  easily  computed,  solving  it  is  par¬ 
ticularly  difficult.  In  practice,  we  find  {z  j}  by  solving 
a  simpler  minimization  problem  introduced  below. 

To  begin,  we  introduce  a  set  of  images,  = 
{u}},  1  <  t  <  K„  where  u*(  is  the  best  approximation 
to  ii£.  It  thus  follows  that 

k 

«/  =  5>«4>»,  (15) 

1=1 

where  for  1  <  i  <  k,  1  <  t  <  K, 

aa  =  §Ju(.  (16) 

By  replacing  ii(  in  the  objective  of  the  problem  (14) 

with  u'l  and  expanding  the  resulting  objective,  we  ar¬ 
rive  at  the  nonlinear  least  squares  minimization 

k  k  2 

min  _  X  L  (a«  -aei(x  i,...,xm))  (17) 

x1ec,,...,xmec7=1(.=1 

k 

X  <1 \>i(xj)aei  =  ue{xj),  l<j<  m,  \<t<K. 

i=  1 

Let  us  denote  a  minimizer  of  this  problem  by 
{z j},  1  <  j  <  m.  We  shall  call  the  z j  as  best  inter¬ 
polation  pixels,  because  these  pixels  are  optimal  for 
the  interpolation  of  the  best  approximations  u  Z.  It  re¬ 
mains  to  describe  the  solution  procedure  for  (17). 


3.3  Solution  Procedure 


We  first  write  the  linear  system  in  (17)  for  a;  = 
[an ,aa]T  into  the  matrix  form  as 

DrDa^  =  D7'd6  1  <£<K,  (18) 

where  df  =  [tq'(xi), . . .  ,ue(xm)]T  and  D  £  M",xi  with 
Dji  =  §i(xj).  Next  let  s  =  [xj, . . .  ,xm]r,  for  1  <  i  < 
k,  1  <£<K,  1  <  q  <  Q  =  kK ,  we  set 

fq(  s)  =  au-aa(sy,  (19) 

m  =  (20) 

Z  9=  1 

The  gradient  and  Hessian  of  the  objective  function 
F( s)  can  thus  be  computed  as 

VF(s)  =  £  /9(s)V/,(s)  =  J(s)rf(s)  ,  (21) 

9=1 


V2F(s)  =  J(s)rJ(s)  +  X  /,(s)V2/,(s)  ,  (22) 

q=  i 

where  J (s)  €  Rex2m,  for  1  <  q  <  Q,  1  <  p  <  2 m. 


Jqp{s) 


d/?(s) 

dxJ  ' 


1  <  j  <  m,d  =1,2. 


(23) 


Hence,  when  the  residuals  /9(s)  are  small,  we  may 
approximately  compute  the  Hessian  in  terms  of  only 
the  Jacobian  matrix  J (s)  as 

V2F(s)  =  J(s)rJ(s)  .  (24) 


To  compute  the  Jacobian  J (s),  we  differentiate  both 
sides  of  (18)  with  respect  to  xj  to  obtain 


<3Dr  7  c)d/,  3E  _ 

a^d/'+D  JkJ 


where  E  =  I) 7 1).  The  partial  derivatives  dd,  /dx'/)r 
dDT/dx^,  and  r)E /dxdm  are  computed  by  finite  differ¬ 
ences.  Note  also  that  x  =  (x1,*2). 

Having  determined  the  gradient  and  the  Hessian, 
we  may  now  use  the  Levenberg-Marquardt  (LM)  al¬ 
gorithm  (Marquardt,  1963)  to  solve  (17).  The  LM 
algorithm  is  very  efficient,  but  it  is  sensitive  to  an  ini¬ 
tial  guess.  Hence  it  is  important  to  start  the  algorithm 
with  a  good  initial  guess.  In  our  implementation,  we 
use  the  empirical  interpolation  method  (Barrault  et  al., 
2004;  Grepl  et  al.,  2006)  to  obtain  an  initial  set  of  in¬ 
terpolation  points  {z'g}  as  follows.  We  first  set 


zf  =  arg  sup  |<|)i  (x)|  .  (25) 

xeE 


Then  for  £  —  2, ... ,  m,  we  solve  the  linear  system 

e-i 

X^(z;S)a;  =  Mz)S).  I<i<^-1,  (26) 

J=l 

for  a7  , 1  <  j  <  t  —  1 ,  and  set 


zf  =  arg  sup 
xez 


j= i 


(27) 


This  set  of  points,  when  used  as  an  initial  guess,  yields 
very  satisfactory  results.  For  further  details  of  the 
empirical  interpolation  method,  we  refer  the  reader 
to  (Barrault  et  al.,  2004;  Grepl  et  al.,  2006). 


3.4  Application  to  Face  Recognition 


We  now  apply  the  BPIM  to  develop  a  fully  automatic 
real-time  face  recognition  system  involving  the  gen¬ 
eration  stage  and  the  recognition  stage.  The  detailed 
implementation  of  the  system  is  given  below: 

1.  Determine  the  dimension  of  the  face  space  k 
and  choose  some  m  (say  m  =  2k),  then  calculate 
4>i , . . .  Note  m  eigenfaces  are  required  to  ob¬ 
tain  the  initial  guess  zf,  1  <  j  <  m. 

2.  Compute  and  store  {z /},  B  =  (CTC)  1  CT .  Re¬ 
call  that  Cji  =  <j),-(Zy),  1  <  i  <  k,  1  <  j  <  m. 

3.  For  a  “gallery”  of  images  VKi  =  {v,},  1  <  i  <  K', 
compute  a,  =  B[v,-(zi),...,v,-(zm)]r,l  <  i  <  K' . 
(Note  Tjf/  can  be  the  same  or  different  from  Uk). 

4.  For  each  new  face  to  be  classified  u,  calculate  its 
interpolant  coefficient  vector  a  from  (13)  and  find 

z'min  =  arg  ^  min  1 1  a  -  a,j  |  .  (28) 

5.  If  1 1  a  —  a.-  II  is  less  than  a  chosen  threshold,  the 
input  image  u  is  identified  as  the  individual  asso¬ 
ciated  with  the  coefficient  vector  a,min.  Otherwise, 
the  image  is  classified  as  a  new  individual. 

The  generation  stage  (steps  1-2)  is  computation¬ 
ally  expensive,  but  performed  only  when  the  training 
set  changes.  Furthermore,  even  if  it  is  necessary  to 
perform  the  generation  stage  due  to  an  update  of  the 
training  set,  we  may  compute  only  the  eigenfaces  and 
reuse  the  best  pixels.  This  can  save  us  some  compu¬ 
tational  time. 

However,  the  recognition  stage  (steps  4-5)  is  very 
inexpensive.  The  calculation  of  a  takes  0(mk).  Note 
further  that  the  problem  (28)  is  the  nearest-neighbor 
search  which  can  be  solved  in  0(kK'°-25)  time  with 
the  storage  of  0(kK'  +  K'XogK')  (Andoni  and  Indyk, 
2006).  Hence,  if  K'  is  in  order  of  0(k4)  or  less,  the 
computational  cost  is  only  0{k2).  This  is  usually  the 


case  even  for  large-scale  applications;  for  example, 
for  a  training  database  of  104  images,  one  would  need 
more  (or  many  more)  than  10  eigenfaces  to  achieve 
acceptable  recognition  rates. 

In  summary,  the  operation  count  of  the  recogni¬ 
tion  stage  is  about  0(mk).  The  computational  com¬ 
plexity  of  our  system  is  thus  independent  of  N.  As 
mentioned  earlier,  the  complexity  of  PCA-based  al¬ 
gorithms  is  at  least  0(Nk).  Our  approach  leads  to 
a  computational  reduction  of  N/m  relative  to  PC  A. 
Since  m  is  typically  much  smaller  than  A,  significant 
savings  are  expected.  The  savings  per  image  classifi¬ 
cation  certainly  translate  to  real-time  performance  es¬ 
pecially  when  many  face  images  need  to  be  classified 
simultaneously. 

However,  some  applications  of  face  recognition 
may  regard  the  recognition  quality  more  importantly 
than  the  computational  performance.  Therefore,  in 
order  to  be  useful  and  gain  acceptance,  our  ap¬ 
proach  must  be  tested  and  compared  with  existing  ap¬ 
proaches,  particularly  here  with  the  PCA. 


4  EXPERIMENTS 

4.1  Face  Database 

The  AT&T  face  database  (Samaria  and  Harter,  1994) 
consists  of  400  images  of  40  individuals  (10  images 
per  individual).  The  images  were  taken  at  different 
times  with  variation  in  lighting,  poses,  and  facial  ex¬ 
pressions,  with  and  without  glasses.  The  images  were 
cropped  and  resized  by  us  to  a  resolution  of  74  x  90. 
We  formed  a  training  ensemble  of  400  images  by  us¬ 
ing  200  images  of  the  database,  10  each  of  20  differ¬ 
ent  individuals,  and  including  200  mirror  images  of 
these  images  (Kirby  and  Sirovich,  1990). 

The  testing  set  contains  the  (200)  remaining  im¬ 
ages  of  20  individuals  not  belonging  to  the  training 
ensemble.  We  further  divide  the  testing  set  into  the 
gallery  of  20  individual  faces  and  180  probe  images 
containing  9  views  of  every  individual  in  the  gallery. 
The  recognition  task  is  to  match  the  probe  images  to 
the  20  gallery  faces.  The  fact  that  the  training  and  test¬ 
ing  sets  have  no  common  individual  serves  to  assess 
the  performance  of  a  face  recognition  system  more 
critically  —  the  ability  to  recognize  new  faces  which 
are  not  part  of  the  face  space  constructed  from  the 
training  set. 

4.2  Results  for  Face  Reconstruction 

We  first  present  in  Figure  4  the  reconstruction  results 
for  a  face  in  the  training  ensemble.  The  BPIM  pro- 
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Figure  4:  The  reconstruction  results  for  a  familiar  face.  The 
BPIM  reconstructed  images  are  placed  at  the  top  row  for 
k  —  40,80, 120, 160  (from  left  to  right)  and  m  =  2k.  The 
PCA  reconstructed  images  are  placed  at  the  second  row  for 
k  =  40, 80, 120, 160  (from  left  to  right).  The  original  face  is 
shown  on  the  right. 
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Figure  5:  Reconstruction  of  a  familiar  face  (bottom  right) 
from  a  10%  mask  (top  right)  with  only  the  white  pixels. 
The  reconstructed  images  are  shown  at  the  top  row  for  k  = 
40, 80, 120, 160  (from  left  to  right)  and  m  =  2k.  The  PCA 
reconstructed  images  are  shown  at  the  second  row  for  k  = 
40, 80, 120, 160  (from  left  to  right)  with  using  all  the  pixels. 


duces  reconstructions  almost  as  well  as  PCA:  most 
facial  features  captured  by  the  PCA  reconstructed  im¬ 
ages  also  appear  in  the  BPIM  reconstructed  images. 
We  underline  the  fact  that  the  interpolation  method 
requires  less  than  5%  of  the  total  number  of  pixels 
N  =  6660,  but  delivers  quite  satisfactory  results. 

To  illustrate  the  use  of  the  interpolation  approach 
for  reconstructing  a  full  image  from  a  partial  image, 
we  consider  a  face  (in  the  training  set)  shown  at  the 
bottom  right  and  a  mask  shown  at  the  top  right  in  Fig¬ 
ure  5.  This  is  a  relatively  extreme  mask  that  obscures 
90%  of  the  pixels  in  a  random  manner.  Because  the 
masked  face  may  not  have  intensity  values  at  all  the 
best  interpolation  pixels,  we  need  to  define  a  new  set 
of  interpolation  pixels.  To  this  end,  we  keep  the  best 
interpolation  pixels  which  coincide  with  some  of  the 
white  pixels  of  the  masked  face  and  replace  the  re¬ 
maining  best  pixels  with  the  “nearest”  white  pixels. 
In  Figure  5,  the  reconstructed  images  using  those  in¬ 
terpolation  pixels  are  compared  with  the  PCA  recon¬ 
structed  images  utilitizing  all  the  pixels.  Although  the 


Figure  6:  Recognition  accuracy  of  PCA  and  BPIM  with  in¬ 
creasing  the  number  of  eigenfaces  k.  Note  that  the  BPIM 
uses  m  =  2k  best  interpolation  pixels. 


interpolation  procedure  does  not  recover  the  original 
face  exactly,  the  construction  is  visually  close  to  the 
“best”  reconstruction. 


Table  2:  Computational  times  (normalized  with  respect  to 
the  time  to  recognize  a  face  for  k  =  10  and  m  =  20  with  the 
BPIM)  for  the  BPIM  and  PCA  at  different  values  of  k. 


Figure  7:  Recognition  of  corrupted  face  images.  The  10% 
mask  on  the  left  is  followed  by  a  few  faces  which  are  cor¬ 
rectly  recognized  with  using  the  interpolation  procedure. 


4.3  Results  for  Face  Recognition 

We  apply  the  face  recognition  system  developed  in 
Section  3.4  to  classify  the  probe  images.  We  illus¬ 
trate  in  Figure  6  the  recognition  accuracy  as  a  function 
of  k  for  the  BPIM  and  PCA.  As  it  may  be  expected, 
the  BPIM  yields  smaller  recognition  rates  than  PCA. 
However,  as  k  increases,  the  BPIM  gives  recognition 
rates  which  are  quite  comparable  to  those  of  PCA  for 
large  enough  k:  PCA  achieves  a  recognition  rate  of 
74.98%,  while  PBIM  results  in  a  recognition  rate  of 
73.66%  for  k  =  80.  In  many  applications,  the  small 
accuracy  loss  of  only  1.32%  is  paid  off  very  well  by 
the  significant  reduction  of  6660/160(>  40)  in  com¬ 
plexity.  This  is  confirmed  in  Table  2  which  shows 
the  computational  times  for  the  BPIM  and  PCA.  The 
values  are  normalized  with  respect  to  the  time  to  rec¬ 
ognize  a  face  for  k  =  10  and  m  =  20  with  the  BPIM. 
Clearly,  the  BPIM  is  significantly  faster  than  PCA. 
This  important  advantage  is  very  useful  to  applica¬ 
tions  that  requires  a  real-time  recognition  capability. 

Finally,  in  order  to  appreciate  the  power  of  the 
interpolation  approach  for  classifying  corrupted  im¬ 
ages,  we  consider  a  random  chosen  mask  of  10%  pix¬ 
els  shown  in  Figure  7.  Next  to  the  mask,  we  show  a 
few  faces  which  are  correctly  recognized  with  using 
the  interpolation  procedure  when  their  intensity  val¬ 
ues  are  available  only  at  the  white  pixels  of  the  mask. 
Note  here  that  the  interpolation  pixels  are  chosen  in 
the  same  way  as  before. 


5  CONCLUSION 

We  have  presented  a  best  points  interpolation 
method  for  the  reconstruction  and  recognition  of  face 
images.  The  method  gives  very  good  reconstruction 
of  face  images.  Therefore,  the  method  has  significant 
potential  to  be  used  as  an  alternative  to  PCA  for  data 
reconstruction.  It  is  important  to  note  that  PCA  uses 
full  knowledge  of  the  data  in  the  reconstruction  pro¬ 
cess.  In  contrast,  the  BPIM  uses  only  partial  knowl¬ 
edge  of  the  data.  Therefore,  the  BPIM  is  very  useful 
to  the  restoration  of  a  full  image  from  a  partial  image. 

We  have  also  developed  a  fully  automatic  real¬ 
time  face  recognition  system  based  on  the  BPIM.  The 
system  is  shown  to  be  able  to  recognize  corrupted 
images.  Moreover,  the  computational  cost  of  rec¬ 
ognizing  a  new  face  is  only  0(mk),  translating  to  a 
saving  of  N/m  relative  to  PCA  approach.  Typically, 
since  N  is  0(  104)  and  m  is  (9(102),  this  implies  two 
orders  of  magnitude  less  expensive  computationally 
than  PCA.  As  confirmed  in  Figure  6  and  Table  2,  the 
system  is  significantly  faster,  while  yielding  a  compa¬ 
rable  recognition  rate  for  large  enough  k,  than  a  stan¬ 
dard  PCA-based  system.  The  significant  reduction  in 
time  should  enable  us  to  tackle  very  large  problems. 
Hence,  it  is  imperative  to  test  our  system  on  a  larger 
database  such  as  the  FERET  database.  We  plan  to 
pursue  this  direction  in  future  research. 

The  present  work  may  also  present  an  opportunity 
combining  the  BPIM  with  some  subspace  techniques. 


For  example,  in  the  context  of  ICA,  instead  of  using 
intensity  values  at  all  pixels  one  may  use  intensity  val¬ 
ues  at  only  the  best  interpolation  pixels  to  build  a  ICA- 
based  recognition  system,  thereby  effecting  signifi¬ 
cant  computational  savings.  It  should  be  mentioned 
that  most  presented  ICA-based  algorithms  do  not  per¬ 
form  ICA  directly  on  the  training  ensemble  of  face 
images,  but  on  either  eigenfaces  or  PCA  coefficient 
vectors,  to  reduce  the  heavy  computational  cost.  Al¬ 
though  we  have  not  put  effort  to  investigate  this  di¬ 
rection,  we  believe  that  using  partial  image  pixels  will 
not  reduce  the  recognition  capability  of  ICA-based  al¬ 
gorithms  provided  that  a  sufficient  (small)  number  of 
the  interpolation  pixels  is  used.  Furthermore,  in  ICA 
Architecture  II  (Draper  et  al.,  2003;  Bartlett  et  al., 
2002),  one  may  want  to  input  the  interpolant  coeffi¬ 
cient  vectors  (instead  of  PCA  coefficient  vectors)  to 
ICA.  Similarly,  one  may  choose  to  perform  LDA  on 
the  interpolant  coefficient  vectors  to  reduce  the  com¬ 
putational  burden  considerably. 

Although  we  do  not  claim  that  our  findings  nec¬ 
essarily  have  a  wide  range  of  applications,  we  believe 
that  our  work  could  open  a  new  direction  of  research 
in  face  recognition  and  image  analysis  in  general. 


ACKNOWLEDGEMENTS 

We  would  like  to  thank  Professor  A.  T.  Patera  of 
MIT  for  his  long-standing  collaboration  and  many  in¬ 
valuable  contributions  to  this  work.  The  authors  also 
thank  AT&T  Laboratories  Cambridge  for  providing 
the  ORL  face  database.  This  work  was  supported  by 
DARPA  and  AFOSR  under  Grant  FA9550-05-1-01 14 
and  F49620-03- 1-0439,  and  by  the  Singapore-MIT 
Alliance. 


REFERENCES 

Andoni,  A.  and  Indyk,  P.  (2006).  Near-optimal  hashing  al¬ 
gorithms  for  approximate  nearest  neighbor  in  high  di¬ 
mensions.  In  Proceedings  of  the  47th  IEEE  Sym.  on 
Foundations  of  Computer  Science,  Berkeley,  CA. 

Barrault,  M.,  Maday,  Y.,  Nguyen,  N.  C.,  and  Patera,  A.  T. 
(2004).  An  “empirical  interpolation”  method:  Appli¬ 
cation  to  efficient  reduced-basis  discretization  of  par¬ 
tial  differential  equations.  C.  R.  Acad.  Sci.  Paris,  Serie 
I,  339:667-672. 

Bartlett,  M.  S.,  Movellan,  J.  R„  and  Sejnowski,  T.  J.  (2002). 
Face  recognition  by  independent  component  analysis. 
IEEE  Trans,  on  Neural  Networks,  13:1450-1464. 

Chellappa,  R„  Wilson,  C.  L„  and  Sirohey,  S.  (1995).  Hu¬ 
man  and  machine  recognition  of  faces:  A  survey.  Pro¬ 
ceedings  of  the  IEEE,  83(5):705-740. 


Delac,  K..  Grgic,  M.,  and  Grgic,  S.  (2005).  Independent 
comparative  study  of  pea,  ica,  and  Ida  on  the  feret 
data  set.  International  Journal  of  Imaging  Systems 
and  Technology,  15(5):252-260. 

Draper,  B.  A.,  Baek,  K.,  Bartlett,  M.  S.,  and  Beveridge,  J.  R. 
(2003).  Recognizing  faces  with  pea  and  ica.  Computer 
Vision  and  Image  Understanding,  91(1-2):  1 15—137. 

Etemad,  K.  and  Chellappa,  R.  (1997).  Discriminant  analysis 
for  recognition  of  human  face  images.  Journal  of  the 
Optical  ^Society  of  America  A,  14(8):  1724-1 733. 

Everson,  R.  and  Sirovich,  L.  (1995).  Karhunen-loeve  pro¬ 
cedure  for  gappy  data.  Opt.  Soc.  Am.  A,  12(8):  1657— 
1664. 

Grepl,  M.  A..  Maday,  Y.,  Nguyen,  N.  C.,  and  Patera,  A.  T. 
(2006).  Efficient  reduced-basis  treatment  of  nonaffine 
and  nonlinear  partial  differential  equations.  M2  AN 
Math.  Model.  Numer.  Anal.  Submitted. 

Jain,  A.  K.  and  Li,  S.  Z.  (2005).  Handbook  of  Face  Recog¬ 
nition.  Springer- Verlag  New  York. 

Kanade,  T.  (2005).  Facial  expression  analysis.  In  Proceed¬ 
ings  of  the  2nd  International  Workshop  on  Analysis 
and  Modelling  of  Faces  and  Gestures,  volume  3723  of 
Lecture  Notes  in  Computer  Science,  page  1 .  Springer. 

Kirby,  M.  and  Sirovich,  L.  (1990).  Application  of  the 
karhunen-loeve  procedure  for  the  characterization  of 
human  face.  IEEE  Transactions  on  Pattern  Analysis 
and  Machine  Intelligence,  12:103-108. 

Lu,  J.,  Plataniotis,  K.  N.,  and  Venetsanopoulos,  A.  N. 
(2003).  Face  recognition  using  Ida-based  algorithms. 
IEEE  Trans,  on  Neural  Networks,  14(1):  195-200. 

Marquardt,  D.  W.  (1963).  An  algorithm  for  least-squares 
estimation  of  nonlinear  parameters.  SIAM  J.  Appl. 
Math.,  1 1  ( 1):43 1—444. 

Nguyen,  N.  C.,  Patera,  A.  T.,  and  Peraire,  J.  (2006).  A  best 
points  interpolation  method  for  efficient  approxima¬ 
tion  of  parametrized  functions.  International  Journal 
of  Numerical  Methods  in  Engineering.  Submitted. 

O'Toole,  A.,  Abdi,  H.,  Deffenbacher,  K.  A.,  and  Valentin, 
D.  (1993).  Low-dimensional  representation  of  faces  in 
higher  dimensions  of  the  face  space.  J.  of  the  Optical 
Society  of  America  A,  10(3):405^111. 

Phillips,  P.  J.,  Wechsler,  H.,  Huang,  J.  S.,  and  Rauss,  P.  J. 
(1998).  The  feret  database  and  evaluation  procedure 
for  face-recognition  algorithms.  Image  and  Vision 
Computing,  16(5):295— 306. 

Samaria,  F.  S.  and  Harter,  A.  C.  (1994).  Parameterisation 
of  a  stochastic  model  for  human  face  identification. 
In  Proceedings  of  the  2nd  IEEE  workshop  on  Appli¬ 
cations  of  Computer  Vision,  pages  138-142,  Sarasota, 
Florida. 

Sirovich,  L.  and  Kirby,  M.  (1987).  Low-dimensional  proce¬ 
dure  for  the  characterization  of  human  faces.  Journal 
of  the  Optical  Society  of  America  A,  4:519-524. 

Turk,  M.  and  Pentland,  A.  (1991).  Eigenfaces  for  recogni¬ 
tion.  Journal  of  Cognitive  Neuroscience,  3(1  ):7 1 — 86. 


